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1. Summary 


This project comprised two related research efforts: 

(A) high-level connectionist cognitive modeling, 

(B) local neural circuit modeling. 

The goals of the first effort were to develop connectionist models of high-level cognitive 
processes such as problem solving or natural language understanding, and to understand the 
computational requirements of such models. The goals of the second effort were to develop 
biologically-realistic models of local neural circuits, and to understand the computational 
behavior of such models. 

In keeping with the nature of NASA’s Innovative Research Program, all the work conducted 
under the grant has been highly innovative. For instance, the following ideas, all summarized 
below, are novel contributions to the study of connectionist/neural networks: 

• the temporal-winner-take-all, relative-position encoding, and pattern-similarity associa- 
tion techniques 

• the importation of logical combinators into connectionism 

• the use of analogy-based reasoning as a bridge across the gap between the traditional 
symbolic paradigm and the connectionist paradigm 

• the application of connectionism to the domain of belief representation/reasoning. 

The work on local neural circuit modeling also departs significantly from the work of related 
researchers. In particular, its concentration on low-level neural phenomena that could support 
high-level cognitive processing is unusual within the area of biological local circuit modeling, 
and also serves to expand the horizons of the artificial neural net field. 

As well as numerous publications and presentations, several Master’s theses and Doctoral 
dissertations were partially supported by the grant. The grant was also part of the impetus 
behind the production of three books in a series edited by Bamden for Ablex Publishing Cor- 
poration. In addition, a small business was set up, and a patent filed. 


i 
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2. Achievements on (A): High-Level Connectionism 


2.1. Winner-Take-All Networks 

Bamden created and investigated the “temporal-winner-take-all” (TWTA) mechanism for 
achieving selection of units/tasks in neural networks. The mechanism is described in: 
Bamden, Srinivas & Dharmavaratha (1990); Bamden & Srinivas (in press); Srinivas & 
Bamden (1990); and Srinivas 1991. 

The TWTA mechanism exploits temporal differences among the arrival times of signals on 
connections as the means for making selections, as opposed to using activation differences as 
in the various conventional types of winner-take-all (WTA) mechanisms widely used in neural 
networks. The TWTA mechanisms has several advantages over conventional WTA tech- 
niques. In particular: 

• it does not require a slow relaxation process; 

• it does not require a large number of extra connections; 

• it does not require exact comparison operations between activation levels; 

• it does not require as much in the way of parameter adjustments to cope with different 

numbers of contenders; 

• and it inherently provides an indication of when it has finished the selection process. 

The TWTA mechanism requires several rounds of contention in general. However, the 
expected number of rounds is merely logarithmic (roughly speaking) in the number of con- 
tending units. 

The TWTA mechanism is an important element in Bamden’s neural net framework, Conposit. 
It represents a particular instance of our approach of establishing the relevance of detailed, 
low-level neural effects to high-level information processing. In fact, as discussed below, the 
mechanism also played an important role in the work on goal (B) of our research (Local 
Neural Circuit Modeling). 

TWTA is especially well suited to cases where an arbitrary selection from among some con- 
tending units or subnetworks must be made. This is a common type of selection operation in 
the Conposit system. 

A detailed unit-level program for simulating the TWTA mechanism was written and tested. It 
was validated by comparison with the results of a detailed mathematical analysis of the sys- 
tem by Srinivas and Barnden. 
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As part of our investigation of TWTA, we conducted a simulation study of various WTA 
mechanisms described in the literature. The experiments confirmed our expectations that con- 
ventional WTA mechanisms are subject to important limitations such as instability, slow con- 
vergence, difficulty in detecting termination, and over-sensitivity to the number of contending 
units. 

The experiments on conventional WTA and on TWTA formed the backbone of Srinivas’ 
Ph.D. dissertation. The TWTA mechanism also played an important role in the Master’s 
theses of two other research assistants, Randall Findley and Mark DeYong, and in DeYong’s 
Ph.D. dissertation. 


2.2. Rule-Based Reasoning Version of the Conposit Framework 

Under the grant, major developments were completed on a structured (though not classically 
localist) connectionist framework for connectionist rule-based reasoning. The culmination of 
this work was a system called Conposit/SYLL. This system implements syllogistic reasoning 
through the manipulation of Johnson-Laird mental models. The intention was not to extend or 
test the Johnson-Laird approach as a psychological theory. Rather, the goal was to verify and 
demonstrate the capabilities of some unusual connectionist techniques for representing com- 
plex information structures. These techniques are Relative-Position Encoding and Pattern- 
Similarity Association. The system also demonstrates the application of the Temporal- 
Winner-Take-All mechanism for selection among contenders (see above). One broad issue 
raised is the way in which the particular nature of the these techniques affects what it is 
natural to do at the symbolic level of description. In particular, the techniques predispose the 
system towards random as opposed to pre-ordered sequencing of subtasks, and towards associ- 
ative linking of symbolic structures as opposed to use of explicit linking constructs. The sys- 
tem is elaborate, but this is because it is designed for procedurely elaborate reasoning tasks. 
The mental model processing achieved is one of the most complex types of symbolic process- 
ing to be tackled in connectionism. 

Conposit/SYLL is described in Bamden (forthcoming, b). Other papers on the Conposit 
framework are Bamden (1989), Bamden (1991a), and Bamden & Srinivas (1991). 


2.3. ABR-Conposit: Analogy-Based Version of Conposit 

The Conposit framework was later developed in the direction of case-based reasoning and 
analogy-based reasoning, leading to a system called ABR-Conposit (initially called CBR- 
Conposit). One motive was to correct Conposit’s main deficiency, namely its lack of learning 
facilities. However, an equally important motive was the goal of gaining both the standard 
advantages of connectionism and those of symbolic systems (without adopting hybrid 
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symbolic/connectionist systems). 

We adopted fully connectionist systems that support analogy-based reasoning as a novel way 
of achieving that goal, at least in the realm of high-level cognitive processing. This domain 
includes common-sense reasoning and the semantic/pragmatic aspects of natural language pro- 
cessing. ABR-Conposit, purely by being analogy-based, gains forms of graceful degradation, 
representation completion, similarity-based generalization, learning, rule-emergence and 
exception-emergence. The system therefore gains advantages commonly associated with con- 
nectionism, although the precise forms of the benefits are different. At the same time, 
through being fully connectionist, the system also gains the traditional connectionist variants 
of those advantages, as well as gaining further advantages not provided by analogy-based rea- 
soning per se. And, because the system is in part an implementation of a form of symbolic 
processing, it preserves the flexible handling of complex, temporary structures that are well 
supported in traditional artificial intelligence and which are essential for high-level cognitive 
processing. 

Although the system has not yet been fully implemented, headway was made on the connec- 
tionist implementation of the long-term memory of source analogs (cases) central to any 
analogy/case-based reasoning system. Also, we partially implemented a novel connectionist 
mechanism for matching two complex symbolic structures with each other. Such matching is 
also central to analogy/case-based reasoning systems. Our matching mechanism rests on the 
use of compressed encodings (reduced representations) of the symbolic structures. (These 
encodings had already been added for independent reasons to the rule-based Conposit frame- 
work, at a late stage of its development.) The matching is essentially a relaxation process, but 
instead of working by means of mutually influencing, scalar activation values, it works by 
means of mutually influencing, vector activation values, namely the compressed encodings. 
Thus, the compressed encodings involved in the two structures being matched are gradually 
adjusted during matching. To our knowledge, this is an unprecedented way of exploiting con- 
nectionist compressed encodings. 

The compressed encodings are also the crux of the process for retrieving source analogs from 
the long-term memory. The compressed encodings allow retrieval to be sensitive to complex 
symbolic structures, not just to individual atoms in those structures. 

The ABR-Conposit approach is described in Bamden & Srinivas (1992a), Barnden (in press, 
b), and Bamden (in press, c). 


2.4. Variables, Logical Combinators and Connectionism 

Connectionist attention to variables has been too restricted in two ways. First, it has not 
exploited ways of doing without variables in the symbolic arena. One variable-avoidance 
method, that of logical combinators, is particularly well established there. Secondly, the 
attention has been largely restricted to variables in long-term rules embodied in connection 
weight patterns. However, short-lived bodies of information, such as sentence interpretations 
or inference products, may involve quantification. Therefore, the question arises of whether 
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short-lived activation patterns need to achieve the effect of variables. 

We have analyzed some benefits and drawbacks of using logical combinators to do without 
variables in short-lived encodings while not losing any expressive power. One benefit is that 
they make encodings and transformations of them qualitatively more uniform and easier to 
devise. The uniformity stands also to facilitate parallelism. Another benefit is that in certain 
circumstances combinators lend extra efficiency to mechanisms that match short-lived encod- 
ings with each other. Such matching is important in high-level cognition. The main drawback 
is that combinators can make symbolic expressions bigger. 

The work on this topic has so far been mainly theoretical (although we did implement a small 
prototype of a connectionist combinator-based system). The work is reported in Barnden & 
Srinivas (1990) and Barnden & Srinivas (1992b). 


2.5. Connectionism and Belief Processing 

The systematicity and structure-sensitivity of high-level cognition, notably common-sense rea- 
soning and natural language understanding, are widely recognized to present a challenge to 
connectionism. However, there is an aspect of systematicity and structure-sensitivity that has 
not been adequately addressed by either side of the symbolicist/connectionist debate. A system 
must have a way of embedding reasoning within various types of context. Notably, a system 
must be able to reason within the context of another agent’s beliefs. 

We analyzed some of the embedding aspect of systematicity, concentrating on the case of 
belief. We identified a considerable challenge that it presents to connectionist systems that 
work by learned holistic processing of compressed encodings (reduced representations). Such 
systems have been shown to have considerable promise for providing systematicity and 
structure- sensitivity while avoiding a straightforward implementation of traditional symbol 
manipulation. Our work leads to a more balanced view of that promise while being sym- 
pathetic to the paradigm. 

We have also analyzed some technical aspects of the representation, in connectionist cognitive 
agents, of the propositional attitudes (beliefs, etc.) of other agents. This work has been 
focused mainly on connectionist systems that do not directly implement symbolic representa- 
tions. One prominent way of symbolically representing attitudes is through meta- 
representational schemes. These have representational expressions that themselves refer to 
representational expressions. Meta-representation is one of the most expressively powerful 
symbolic approaches for attitude representation. Therefore, we have the question of whether 
non-implementational connectionist systems could use an analogous approach. 

Unfortunately, it is not straightforward to devise a plausible analogy to symbolic meta- 
representation. We have looked at three main possibilities: (i) the representational activation 
patterns of the non-implementational connectionist system refer to the system’s own activation 
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patterns; (ii) the activation patterns refer to formal symbolic expressions; and (iii) the activa- 
tion patterns refer to natural-language expressions. We claim that possibility (iii) avoids some 
of the problems of (i) and (ii), and also has benefits that are independent of specifically con- 
nectionist considerations. The analysis highlighted the point that even a non-implementational 
connectionist system must be able to make inferences about complex symbolic constructs such 
as logic expressions and natural language phrases, even though it does not make inferences 
with them. 

The work on connectionism and beliefs is reported in Bamden (1991b, 1992a, b,c,d), Barnden 
(in press, a), and Bamden (forthcoming, a). 


3. Achievements on (B): Local Neural Circuit Modeling 

The research effort in low-level, local neural circuit modeling was focussed on the_develop- 
ment of electronic circuit models of processing elements that simulate neurons, components of 
neurons, or small groups of neurons. 

Our first modeling goal was to develop realistic models of relatively simple pattern-generating 
circuits such as are found in invertebrates. We developed circuit models of all of the major 
components of a generic spiking neuron: excitatory and inhibitory chemical synapses, the den- 
dritic tree, the soma, the axon hillock, and the axon. The behavior of these mo del s under a 
range of conditions was simulated using the general analog circuit simulation program SPICE, 
version 3c. The component models were used as building blocks to construct circuit models 
of central pattern generators, logic elements, and Temporal-Winner-Take-All decision net- 
works. 

The electronic circuit models employ conventional CMOS, and in some cases BiCMOS, 
integrated circuit technology, and are intended for fabrication as VLSI devices. The models 
preserve both the pulse shapes and relative timing of both action potentials and post-synaptic 
potentials (PSPs) observed in neurons; however, these are linearly scaled from the mV - ms 
voltage and time domains to the V - ns domain. This scaling substantially decreases the noise 
sensitivity of the models, and allows the use of designs having smaller numbers of devices, 
while increasing the speed with which computations can be performed by approximately six 
orders of magnitude. 

We developed a set of second generation models that allowed all of the major model parame- 
ters - the amplitudes and widths of the PSPs and the threshold and timing of action potential 
firing - to be varied adaptively. Both purely CMOS and mixed CMOS - BiCMOS versions of 
these circuits were developed; these have the advantages of easier fabrication and higher 
speed, respectively. These adaptive circuits have allowed us to investigate relatively slow 
forms of learning such as implemented by conventional artificial neural networks, as well as 
rapid forms of neural modulation observed in nervous systems. We also started an investiga- 
tion into the utility of these models for implementing nearest-neighbor classification algo- 
rithms, which have been used successfully for a wide range of pattern recognition, time-series 
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prediction, and motor control tasks. 

The work is reported in the papers by DeYong, Fields and/or Findley in the publication list 
below. 

The Temporal-Winner-Take-All mechanisms is important in both branch (A) and branch (B) 
of the research supported by the grant, and features in DeYong’s thesis and dissertation. It 
provides a salient instance of the way in which low level neural phenomena, such as the fine 
signal timing differences exploited by temporal-winner-take-all, can make an important contri- 
bution to high-level connectionist modeling. 


4. Personnel 

The project personnel were as follows: 

John Barnden, PI. 

Christopher Fields, PI. 

Imre Balogh, Graduate Research Asst, Ph.D. student in Computer Science. 

Mark DeYong, Graduate Research Asst, M.S. student in Electrical and Computer Engg. 
Randall Findley, Graduate Research Asst, M.S. student in Electrical and Computer Engg. 
Kankanahalli Srinivas, Graduate Research Asst, Ph.D. student in Computer Science. 
Heather Pfeiffer, Graduate Research Asst, Ph.D. student in Computer Science. 

Thomas Eskridge, Graduate Research Asst, Ph.D. student in Computer Science. 


5. Landmark Events 

(1) Mark DeYong and Randall Findley, research assistants on the project, obtained their 
Master’s degrees from the Department of Electrical and Computer Engineering in Spring 
1991. DeYong later went to receive his Ph.D. on an extension of the master’s work. The 
three theses were on topics central to effort (B) and were based directly on their work as 
Research Assistants on the project. 

(2) Kankanahalli Srinivas obtained his Ph.D. degree from the Department of Computer Sci- 
ence in Spring 1991. His dissertation was on the topic of winner- take- all networks, which is 
central to effort (A) and also connects to part of effort (B). The dissertation was based 
directly on his work as a Research Assistant on the project. The abstract is appended to this 
report. 

(3) Imre Balogh is close to obtaining his Ph.D., for research based on the 
connectionism/beliefs work described under (A) above. Balogh was a research assistant on the 
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grant. 

(4) The grant research was partially responsible for leading to the production of a book 
(Bamden & Pollack) consisting of a collection of chapters on the area of effort (A). Most of 
the papers were written especially for the book. 

(5) The book mentioned in (4) is the first volume in a series entitled Advances in Connection- 
ist and Neural Computation Theory. Bamden is the Series Editor. The second and third 
volumes are currently in press, for publication near in the Fall of 1993. See the Holyoak & 
Bamden and Bamden & Holyoak items in the publication list below. The volumes are tightly 
linked, and are edited collections of papers on the question of applying connectionism to 
analogy-based reasoning, case-based reasoning and metaphor. All the chapters were written 
especially for the volumes. 

(6) In May 1992, a patent was filed by New Mexico State University on work resulting from 
aspect (B) of the grant research. The patent title is “Asynchronous Temporal Neural Process- 
ing Element.” The patent is pending. 

(7) Mark DeYong (a research assistant on the grant) set up a small business. Intelligent Rea- 
soning Systems, as a spin-off of aspect (B) of the grant research. The business is currently 
funded by three SBIR grants — one from the NSF and two from the Department of Defense. 
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